Towards Research Performance in the Humanities

THISPAPER DESCRIBES A general methodology for developing bibliometric performance indicators. Such a description provides a framework or paradigm for application-oriented research in the field of evaluative quantitative science and technology studies, particularly in the humanities and social sciences. It is based on our study of scholarly output in the field of Law at the four major universities in Flanders, the Dutch speaking part of Belgium. The study illustrates that bibliometrics is much more than conducting citation analyses based on the indexes produced by the Institute for Scientific Information (ISI), since citation data do not play a role in the study. Interaction with scholars in the fields under consideration and openness in the presentation of the quantitative outcomes are the basic features of the methodology. Bibliometrics should be used as an instrument to create a mirror. While not a direct reflection, this study provides a thorough analysis of how scholars in the humanities and social sciences structure their activities and their research output. This structure can be examined empirically from the point of view of its consistency and the degree of consensus among scholars. Relevant issues can be raised that are worth considering in more detail in followup studies, and conclusions from our empirical materials may illuminate such issues. We argue that the principal aim of the development and application of bibliometric indicators is to stimulate a debate among scholars in the field under investigation on the nature of scholarly quality, its principal dimensions, and operationalizations. This aim Henk F. Mocd, Centre for Science and Technology Studies ( C W T S ) , Leiden University, P.O. Box 9555,2300 RB Leiden, the Netherlands Mark Luwel, Science and Innovation Admini?tration, Ministry of the Flemish Community, Boudewijnlaan 30, Brussels, Belgium A.J.Nederhof, Centre for Science and Technology Studies (CWTS), Leiden University, P.O. Box 9555,2300 RB Leiden, the Netherlands LIBRARY TRENDS, Vol. 50, No. 3, Winter 2002, pp. 498-520 02002 The Board of Trustees, University of Illinois MOED, LUWEL, 8c NEDERHOF/RESEARCH PERFORMANCE 499 provides a criterion of “productivity” of the development process. We further contend that librarians are not infrequently requested to provide assistance in collecting data related to research performance assessments, and that the methodology described in the paper aims at offering a general framework for such activities, and can be used by librarians as a line of action whenever they become involved. 1.INTRODUCTION The study presented in this paper focused on the fundamental questions: How does one recognize a “good” scholar? How does one recognize an “important” scholarly contribution? The approach adopted in this study can be defined as bibliometric. It aims at identifjmg characteristics of scholarly publications that can validly be assumed to reflect the “quality” or “importance” of a scholar or a scholarly work. Therefore, a first answer to the question “How does one recognize a ‘good’ scholar?” is: One should examine his or her scholarly publications. In other words, in a bibliometric approach, it is assumed that important contributions to scholarly progress are sooner or later communicated in scholarly publications. This is considered to be a universal characteristic of scholarly development in natural sciences, life sciences, social sciences, and humanities. A bibliometric approach is a quantitative approach. It attempts to calculate statistics of quantitative aspects derived from scholarly publications. Bibliometric indicators result from the statistical analysis of bibliographic information retrieved from the scholarly literature. This determines both their strength and their limitations. The strength of the bibliometric method is that, once established, it can be applied in a uniform or objective manner, eliminating the influence of subjective or personal factors. On the other hand, being a statistical method, it cannot take into account all particularities or special features of the objects to be assessed. As a consequence, bibliometric data should always be applied in combination with qualitative knowledge about the scholars involved and the subdisciplines in which they are active. Bibliometric indicators have been successfully applied in many subdisciplines in the natural and life sciences. Data from the Science Citation Index (SCI) ,produced by the Institute for Scientific Information (ISI) ,play an important role in analyses of research performance in these subdisciplines (e.g., van Raan, 1996;Van Den Berghe et al., 1998). Thus far, the social sciences and humanities have not often been subjected to such analyses. At the same time, the academic authorities of many universities have expressed the need to obtain an insight into the research performance of all faculties and in all fields of scholarship. Fundamental differences exist between the natural and life sciences, on the one hand, and the humanities and social sciences, on the other hand, with respect to the research object, the methodologies applied, and the 500 LIBRARY TRENDS/WINTER 2002 structure of scholarly communication. As a result, those who are involved in the development of performance indicators for the humanities and social sciences are confronted with the following situation. Firstly, they need to develop methodological tools to assist evaluation agencies or policymakers in carrylng out their tasks, in the same way that the current SCI-based methodologies provide supplementary research assessment tools in the natural and life sciences. Secondly, this methodology should take into account the characteristics of the field of scholarship, the nature of the scholarly research object, and particularly the communication practices among scholars and the structure of the communication system in their fields. A study dealing with these challenges can indeed be considered an endeavour. This paper attempts to describe a general methodology for developing bibliometric performance indicators. Such a description provides a framework or paradigm for application oriented research in the field of evaluative quantitative science and technology studies, particularly in the humanities and social sciences. It is based on a study on scholarly output in the field of Law at the four major universities in Flanders, the Dutch speaking part of Belgium. The background, setup, and methodological framework are presented in Section 2. It is followed by a concise review in Section 3 of earlier studies on research performance in this field of scholarship. Section 4 presents a number of characteristic outcomes of the study. Its principal aim is to illustrate the methodology outlined in Section 2. A detailed overview of the study is presented in a research report by Luwel et al. (1999).The study illustrates that bibliometrics is much more than conducting citation analyses based on the IS1 citation indexes, as citation data do not play a role in this study. Finally, Section 5 gives a critical discussion of the methodology, in the light of the experiences collected in the study. This discussion includes a short overview of the comments of scholars and of the followup of our study, and summarizes the main features of our methodology. 2. BACKGROUND, SETUP, AND METHODOLOGICAL FRAMEWORK 2.1 Background and Setup of the Study The study presented in this paper was a pilot study commissioned by the Flemish Inter-University Council (VLIR) for developing a methodology to assess research performance in the social sciences and humanities. The disciplines selected by the VLIR for this study were Law and Linguistics. This paper discusses only the study on Law. The Catholic University of Leuven, the University of Gent, the Flemish-speaking Free University of Brussels, and the University of Antwerp decided to participate in the study, which was partially funded by a grant approved by the Flemish Minister-President, who is also in charge of science and technology policy. MOED, LUWEL, 8C NEDERHOF/RESEARCH PERFORMANCE 501 At the start of the project, in early 1997, a project team was set up, consisting of the research staff of the VLIR, researchers of the Centre for Science and Technology Studies (CWTS) at Leiden University (the Netherlands), and a staff member of the Science and Innovation administration of the Ministry of the Flemish Community. The activity of the project team was supported by a university expert group, set up for each of the two disciplines, and composed of senior academic staff members of the departments of Law and Linguistics at the four participating universities. The expert groups assisted the project team during the elaboration of the project, and played an active role in their respective universities in a series of activities, such as data collection, the develop ment of classification systems, and commenting on drafts of the final report. The first stage of the project work plan was a clear boundary setting of the two disciplines. For the Law faculties, this operation was relatively simple. Academic staff data were extracted from the universities’ central administration databases, including year of birth, gender, starting and ending date of their appointments, rank, length of appointment (e.g., 40 percent, that is, two days a week), funding source, and year of Ph.D. granting. In a subsequent phase, a questionnaire was prepared to collect quantitative data. It was sent to all researchers, both junior and senior, active in one of the four Flemish universities at the end of the year 1996. The members of the two expert groups were of the opinion that the study should not be limited to research activities only, but that all academic activities should be taken into account, analyzing also the fraction of work dedicated to research. The most important data, analyzed in this paper, were lists of publications. In tandem with the analysis of the first questionnaire, a second questionnaire was elaborated, again in collaboration with the expert groups, in order to collect more qualitative information on leading publications, journals, publishers, and Flemish scholars in the two disciplines. The main objective was to obtain insight into scholars’ perceptions on scholarly work quality, and to assess to what extent the scholars’ opinions corroborated the outcomes of the quantitative indicators. For Law, this questionnaire was sent to professors working at Flemish, Dutch, and Belgian French-speaking universities,aswell as to Belgian senior magistrates. Respondents were asked to indicate “outstanding,” “good, yet not outstanding,” and “less good” journals, and the names of Flemish scholars whose work is currently very important to their subdiscipline. A draft report was sent to and discussed with the two expert groups, and the results of these discussions were incorporated into the final version of the report. 2.2 Methodological Framework Our study was primarily a methodological one. Rather than making comparative evaluation statements on research performance at the four universities, it explores methods to provide a clear insight into scholarly research 502 LIBRARY TRENDS/WINTER 2002 activities and proposes indicators for measuring relevant aspects of scholarly performance. It examines the validity of such indicators and explores the type of data needed to construct them, taking into account the availability and reliability of such data. In this study, the participants were confronted with the problem of lack of standardization in the publication practices of Law scholars. This is a problem in many subfields in the social sciences and humanities. By contrast, from interviews with scientists in molecular biology, it appears that this subfield has a strong consensus of how research materials should be published, and which journals are the most prestigious. All important research output is published in English, in international journals. There are some five to tenjournals that are generally acknowledged as “top” journals (e.g., Van Den Berghe et al., 1998).Many publication lists of scientists have a standard format, and small contributions, such as meeting abstracts or editorials, are not even listed. However, in the humanities, and particularly in Law, important contributions are often published in commemorative books with a narrow circulation. There are many types of publications, and publication lists of scholars are often not ordered by type. Publications are often in the mother language, and many activities have an applied nature. In our study, a thorough analysis of the publication output was conducted. The quality of the bibliographic information was assessed. The scholars providing the publication data added several types of additional information to each publication, using classification systems of types of publications and subdisciplines. This additional information was examined carefully, from the point of view of its accuracy, embedded structural relationships, the degree of consensus among scholars, and the extent to which differences existed among subdisciplines. Typical examples of more specific research questions were: How important is the role of books in the communication among scholars? How can one reliably measure the number of books published by a scholar during a given time period? How important is the role ofjournals? Are there differences among subdisciplines? In which ways do publications classified by juridical scholars as “substantial contributions” differ from “small contributions”?Do the two types of publications reveal different bibliometric characteristics? How consistently was this distinction made among scholars? Are there any criteria to discriminate between scholarly journals and journals of a more applied nature? Assuming that, in the field of Law, it is appropriate to make a distinction between a “scholar” and a “practitioner,” are there any bibliometric indicators that can be used in helping to discriminate between the two types? What is the perception of Flemish, Belgan French-speaking, and Dutch scholars on the quality of individual Law journals? Were there significant differences between the perceptions of Flemish scholars and those of their MOED, LUWEL, & NEDERHOF/RESEARCH PERFORMANCE 503 colleagues abroad? What does a tentative ranking of journals look like, based on their quality, as perceived by scholars, and their international visibility? What is the position of Flemish journals in such ranking? To what extent do actual publication strategies of Flemish scholars conform to a “quality standard” as expressed in their own ratings of scholarly journals? 3. A CONCISEREVIEW OF EARLIERSTUDIESON RESEARCH PERFORMANCE IN LAW Several authors have addressed the measurement of research performance in the social sciences and humanities from a general perspective (Cole, Cole, & Dietrich, 1978; Garfield, 1979, 1986; Cole, 1983; Nederhof et al., 1989; Kyvik, 1989; Finkenstaedt, 1990; Nederhof & Zwaan, 1991; Nederhof & Noyons, 1992; Hemlin, 1996; Hemlin & Gustafsson, 1996; Wood, 1998). Law, in particular, has been called “the birthplace of citation study” (Shapiro, 1992, p. 339). For instance, according to Shapiro, in 1894 a table showing the comparative citation frequency of the Federal, English, and State decisions was produced in Boston, and in 1817 a first count of the volume of English Law reports was made. Notwithstanding this very early start, a literature search revealed very few articles referring to the measurement of scholarly performance in Law (cf. Justiss, 1993). Swygert & Gozansky (1985) studied the productivity of 1,950 U.S. senior Law faculty members (full-time appointed, full professors) by examining their publications in the Legal Resource Index (LRI) and the database of the Online Computer Library Centre, Inc. (OCLC) during three and four years, respectively. The coverage by LRI (mostly articles and book reviews) and OCLC (book titles) was deemed nearly complete. Swygert & Gozansky (1985, p. 378) included a wide variety of publications: “Articles, books, book reviews, casebooks, teacher manuals, practice manuals, textbooks, monographs, treatises, supplements and compilations, as well as edited and co-authored works,” provided the entry was five pages or longer in length. Testimonials, obituaries, reports or proceedings, bibliographies, newspaper columns, recordings and any title labelled “bar review notes” were not included. The results showed that 44 percent of the faculty members had no publications whatsoever, while 65 percent had no more than 1publication. Only 15 percent had four or more publications. The mean number of publications was 1.5 per faculty member over a period of three to four years, or less than 1publication in two years. A school or university published on average about 18 items, with the faculty at Chicago (mean = 5.1 publications per member), Cornell, and New York University ranking highest in average productivity, followed by Berkeley, Stanford, Yale, and Harvard (all with 3.0 publications or more per faculty member). Schools with less senior faculty members were less productive on average (as indicated by the Pearson correlation coefficient, r = 0.53). The authors assumed that smaller-sized 504 LIBRARY TRENDS/WINTER 2 0 0 2 schools had a higher teaching load per head. It should be noted that the productivity of faculty members of lower age (the mean was fifty-one years), and striving for a tenured position, may well be considerably higher. Other less extensive studies analyzed the productivity of institutions by recording author affiliation in journals of high prestige (e.g., Ellman, 1983; Sorensen, 1994). Two publications related to research performance assessment in juridical research are of particular interest to the study presented in this report. The first is the November 1996 final report of the Inter-University Committee of the Flemish Faculties of Law, entitled “The Assessment of Performance in Juridical Research.” This report presents a classification of scholarly publications in the field of juridical research. In our study, a classification scheme was applied that is principally based upon that of this committee. Therefore, our study can be viewed as a first large-scale experiment with this classification system. According to the Inter-University Committee of the Flemish Law Faculties, the published book ranks first in the scholarly juridical publication output. A book is viewed as the result of an often individual and personal synthesisof legislation, jurisdiction, and juridical theory in a subdiscipline. A book often reflects continuous, intensive scholarly research, conducted for many years. In the Committee’s view, the same is true for doctoral theses. Therefore, as a rule, a doctoral thesis deserves publication as a book. The Committee also made a distinction between substantial scholarly contributions and scholarly contributions of a limited size, published in accepted scholarly journals, anniversary volumes, seminar reports, and collective works. Typical examples of the first type are: A leading article, a review on jurisdiction, or a thorough annotation. A short annotation, a thorough book review, or an intervention as panel member or participant in a conference are examples of scholarly contributions of a limited size. The Committee did not succeed in developing a classificationof scholarlyjournals in terms of their quality or reputation. The main impediment to such a ranking was that most Law journals show large variations in the quality of the papers published. In addition, some subdisciplines are covered by a limited number of nationaljournals only, for which no definitive ranking could be made. A second report that bears a high relevance to our study is the “Quality Assessment of Research-Rechtsgeleerdheid,” published in April 1996, by the Review Committee on Juridical Research, set up by the Association of Universities in the Netherlands (VSNU).The report presents an assessment of research activities in Law at universities in the Netherlands. In the publication output assessment, the Dutch VSNU Committee applied several criteria to identify the most valuable scholarly works in the mass of publications listed. Firstly, the Committee made a distinction between first editions and later editions of singleor multiauthored books. Interestingly, MOED, LUWEL, & NEDERHOF/RESEARCH PERFORMANCE 505 this aspect is not mentioned in the report of the Committee of the Flemish Law Faculties. Secondly, in order to discriminate between substantial and small scholarly contributions, the VSNU Committee took into account the publications length as reflected in the number of pages. Publications with a length of more than five pages were regarded as “substantial” contributions. For each research programme to be assessed, the Committee regarded the number of singleor multiauthored books (first editions only), doctoral theses, and articles ofwhich the number of pages exceeded five, as the most significant productivity measure. In addition, the total number of publications (of all types) was determined. In its final report, the VSNU Committee expressed the need for clear guidelines and criteria for selecting and structuring the information on publication output. Such criteria should first of all specify the type of publications to be included in a performance or quality assessment. In addition, the Committee stressed that attempts should be made to distinguish between “genuine” scholarly contributions, on the one hand, and informative publications primarily aimed at providing social services, on the other. Genuine scholarly publications conform to criteria of methodological soundness, thoroughness, and significance. In the Committee’s view, it is the first category of publications that distinguishes between ajuridical scholar and a practitioner or a professional legal expert. Academic scholars should be primarily evaluated according to their contribution to scholarly progress, rather than to their practical activities. The relationship between juridical research and practice is also addressed in the report by the Inter-University Committee of the Flemish Law Faculties. This committee stated thatjuridical research primarily serves the practice, a basic characteristic that creates difficulties in distinguishing between fundamental and applied juridical research. 4. RESULTS The core of our analyses can be denoted as bibliometric, and related to publications. In this section, the main findings are summarized related to the Flemish juridical scholars’ publication output. A detailed account is given in the research report by Luwel et al. (1999). 4.1 Units of Analysis We agree with McGrath’s (1996) statement that it is crucial in any b i b liometric study to define carefully its units of analysis.. This study deals with several units of analysis. The first is the individual publication. The main aspect is the classification of individual publications. In view of the main interest of our study-research performance assessment-an attempt was made to rank the various types of publications according to their importance or size of contribution to scholarly progress. A second unit of analy506 LIBRARY TRENDS/WINTER 2 0 0 2 sis is that of the journal. Journals were classified on the basis of characteristics of the publications included, and also on the basis of judgments by peers obtained from a questionnaire. The distinction between scholarly journals and journals of an applied nature or directed towards a broad audience is a crucial element. A third unit of analysis is the individual scholar. Here, the distinction between “scholar” and “practitioner” is important, based on an analysis of the type of publications made, the type ofjournals used, and also on qualityjudgments obtained from a questionnaire. 4.2 Publication Output: Classzjicntions In the first questionnaire, respondents listed their complete publication output during the 1992-1996 time period. The total number of publications listed amounted to 3,753. All publications were arranged into types. The classification of publications in Law applied in this study is largely based on the 1996 final report of the Inter-University Committee of the Flemish Law Faculties (see Section 3). The classification system is presented in Table 1. In this system, a book is the most important publication, as it is often the reflection of continuous, intensive scholarly research, conducted for Table 1. Classification Scheme of Scholarly Publications in Law, Applied in the